{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4.1 判别函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第三章主要介绍了线性模型在回归问题中的应用。\n",
    "\n",
    "现在我们考虑分类问题：对于一个输入向量 $\\bf x$，我们的目标是将它划分为 $K$ 类中的一类 ${\\cal C}_k$，其中$k=1,\\dots,K$。\n",
    "\n",
    "通常这些类别都是不相交的，因此每个输入只对应于一个类别。这样，我们相当于将输入空间划分为多个决策区域，不同决策区域的边界叫做决策面（`decision surfaces`）。在本章，我们考虑线性分类模型。此时，决策面由输入向量 $\\bf x$ 的线性函数决定，因此是一个输入 $D$ 维空间中的一个 $D-1$ 维的超平面。\n",
    "\n",
    "能被线性决策面完全分开的数据称为线性可分的。\n",
    "\n",
    "对于回归问题，我们的目标值就是我们的预测值；对于分类问题，由于类别是离散的，不能直接预测，我们需要用一些其他方式来表示我们的类别。对于概率模型，最简单的方式是 0-1 表示。例如，对于二类问题，我们用单一目标值 $t\\in\\{0,1\\}$ 来表示类别，$t = 1$ 表示类别 ${\\cal C}_1$，$t = 0$ 表示类别 ${\\cal C}_2$，此时，$t$ 可以看成是样本属于类别 ${\\cal C}_1$ 的概率。\n",
    "\n",
    "对于 $K > 2$ 的情况，1-hot是一种常用的表示方法，即用一个 $K$ 维的 0-1 向量 $\\bf t$ 表示类别，当类别为 ${\\cal C}_j$ 时，除了 $t_j$ 为 1 之外，其他的 $t_k$ 都为 0。例如 $K = 5, j = 2$ 的情况对应于目标向量：\n",
    "\n",
    "$$\n",
    "\\mathbf t = (0,1,0,0,0)^\\top\n",
    "$$\n",
    "\n",
    "同样，我们也可以将 $t_k$ 看成是类别是 ${\\cal C}_k$ 的概率。\n",
    "\n",
    "在线性回归模型中，预测值 $y({\\bf x, w})$ 是一个关于参数 $\\bf w$ 的一个线性函数。最简单的情况为 $y({\\bf x})=f({\\bf w^\\top x} + w_0)$，此时 $y$ 可以为任意一个实数。而在线性分类的模型中，我们的预测值可以看成是概率，因此要限制在 $(0,1)$ 之间。为此，我们将上面的模型一般化，在参数 $\\bf w$ 的线性函数之上，加一个非线性函数 $f(·)$：\n",
    "\n",
    "$$\n",
    "y({\\bf x})=f({\\bf w^\\top x} + w_0)\n",
    "$$\n",
    "\n",
    "函数 $f$ 通常被叫做激活函数（`activation function`）。决策面对应与 $y({\\bf x})={\\rm constant}$，所以这相当于 ${\\bf w^\\top x} + w_0={\\rm constant}$，从而决策面对 $\\bf x$ 是线性的，即使激活函数 $f(·)$ 是非线性的。不过，由于函数对参数 $\\bf w$ 不是线性的，因此计算和优化与线性回归模型不同。\n",
    "\n",
    "在第一章，我们讨论了三种解决分类问题的方法：判别函数，判别式模型 $p({\\cal C}_k|{\\bf x})$，产生式模型 $p({\\bf x}|{\\cal C}_k)$。\n",
    "\n",
    "首先，讨论最简单的线性判别函数。\n",
    "\n",
    "Generalized Linear Model(GLM):\n",
    "\n",
    "(1) 当 f 是 nonlinear function 的时候,GLM 是一个classification model\n",
    "\n",
    "(2) 当 f 是 identity function 的时候， GLM 是一个 regression model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1.1 二类问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于二类问题，最简单的线性判别函数为：\n",
    "\n",
    "$$\n",
    "y({\\bf x})={\\bf w^\\top x} + w_0\n",
    "$$\n",
    "\n",
    "其中 $\\bf w$ 是权重（`weight vector`），$w_0$ 是偏置项（`bias`）。偏置项的相反数也叫阈值（`threshold`）。\n",
    "\n",
    "如果 $y({\\bf x})\\geq 0$，那么 $\\bf x$ 属于 ${\\cal C}_1$ 类；否则属于 ${\\cal C}_2$ 类。此时，我们的决策面为 $y(\\mathbf x)=0$，对应与输入 $D$ 维空间中的一个 $D-1$ 维的超平面。\n",
    "\n",
    "对于这个超平面上的两个点 $\\mathbf x_A,\\mathbf x_B$，由于 $y({\\bf x}_A)=y({\\bf x}_B)=0$，我们有 $\\mathbf w^\\top(\\mathbf x_A-\\mathbf x_B)=0$，从而 $\\bf w$ 与超平面中的任意向量垂直，即是超平面的法向量。\n",
    "\n",
    "根据法向量的性质，设超平面上的某点为 $\\bf x$（意味着 $y({\\bf x})=0$），超平面到原点的距离为\n",
    "\n",
    "$$\n",
    "\\frac{\\bf w^\\top x}{\\|\\bf w\\|}=-\\frac{w_0}{\\|\\bf x\\|}\n",
    "$$\n",
    "\n",
    "这样我们就看到，决策面的位置由偏置项 $w_0$ 来决定。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1.2 多分类问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "one-versus-the-rest：用K-1个二分类器来解决一个K类别分类任务\n",
    "\n",
    "one-verse-one: 用$\\frac{K(K-1)}{2}$个二分类器来解决一个K类别分类任务，两种分类方法都会造成无法分类的区域\n",
    "\n",
    "single-K-class-discriminant: 由K个线性函数组成的一个K类别判别函数$y_{k}(x)=w_{k}^{T}+w_{k0}$，类k和类别j的决策边界时$y_{k}(x)=y_{i}(x)$也就是超平面${w_{k}-w_{j}}^{T}x+(w_{k0}-w_{j0})=0$,若一个数据点被分配给了k，则$y_{k}(x)>y_{i}(x)$。另外有凸函数的定义可以证明，这种分类函数形成凸空间，不存在无法分类的区域。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1.3 最小平方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$y(x)=\\tilde{\\mathbf W}^{T}\\tilde x$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$E_D(\\tilde{\\mathbf W})=\\frac{1}{2}Tr{(\\tilde{\\mathbf X}\\tilde{\\mathbf W}-T)^T(\\tilde{\\mathbf X}\\tilde{\\mathbf W}-T)}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\tilde{\\mathbf W}=（{{\\tilde{\\mathbf X}}^T\\tilde{\\mathbf X}})^{-1}{\\tilde{\\mathbf X}}^T{\\mathbf T}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最小平方法缺少鲁棒性，因为其对应于高斯条件分布假设下的最大似然法，而二分类向量的概率显然不是高斯分布"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(r\"../\")\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "from prml.features import PolynomialFeatures\n",
    "from prml.linear import (\n",
    "    BayesianLogisticRegressor,\n",
    "    LeastSquaresClassifier,\n",
    "    LinearDiscriminantAnalyzer,\n",
    "    LogisticRegressor,\n",
    "    Perceptron,\n",
    "    SoftmaxRegressor\n",
    ")\n",
    "\n",
    "np.random.seed(666)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_toy_data(add_outliers=False, add_class=False):\n",
    "    x0 = np.random.normal(size=50).reshape(-1, 2) - 1\n",
    "    x1 = np.random.normal(size=50).reshape(-1, 2) + 1.\n",
    "    if add_outliers:\n",
    "        x_1 = np.random.normal(size=10).reshape(-1, 2) + np.array([5., 10.])\n",
    "        return np.concatenate([x0, x1, x_1]), np.concatenate([np.zeros(25), np.ones(30)]).astype(np.int)\n",
    "    if add_class:\n",
    "        x2 = np.random.normal(size=50).reshape(-1, 2) + 3.\n",
    "        return np.concatenate([x0, x1, x2]), np.concatenate([np.zeros(25), np.ones(25), 2 + np.zeros(25)]).astype(np.int)\n",
    "    return np.concatenate([x0, x1]), np.concatenate([np.zeros(25), np.ones(25)]).astype(np.int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAADHCAYAAAD8tCOMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8HNW1wPHf2S5ZsuUC7rhRjY0NNg6994QYQiCUACEE\nAi+QhBRCEgikk0ZJIARCCJA8DDwnEIfeMWCaTXMDDMbGlrstyWqrbef9MSN5Je1KK2mlLTrfz0cf\n7+7MzpyVzx7duXPnjqgqxhhjipMn1wEYY4zpPVbkjTGmiFmRN8aYImZF3hhjipgVeWOMKWJW5I0x\npohZkTemnxCRc0TkqW6+d6mIHJHlkPKeiDwuIufnOo6e6NdFXkRWicgxfbg/FZFdO1geEJE/iMha\nEalz47upr+Iz+aM3clNV/1dVj8tg33eLyC/avHdvVX2hK/sTkfFuztcl5fNVXQw7p1T1RFW9J9dx\n9IQv1wGYVn4IzARmAeuBccBhfR2EiPhUNdbX+zVFq0JVYyIyE3hRRBap6tPZ3IHlbHr9uiXfERH5\nnIi8IyLVIrJARPZJWnaViHwsIrUiskxETk1atquIvCgiNSKyRUQecF+f767yrtuq+VKK3e4PPKSq\n69SxSlXvTdr2viLylrvfB0Tk/uYWl4h8RURebvMZWo4cROSzIvK2iGwXkTUicl3Ses0trgtF5FPg\nOff1A9zPXi0i7yYfrrv7W+nG8omInNPd37XpGhG5SEQ+EpFtIjJPREYlLTtORD5w8+/Pbi5+zV3W\nkiPiuFFENrk5sVhEpojIxcA5wJVunv7XXb/lyEJEvCLyo6TvwCIRGdtZ3Kq6EFgKTE+Kd5SI/EtE\nNrt59M2kZSUico+IVInIchG5UkTWJi1fJSI/EJH3gHoR8XWyvVkistD9vBtF5Ab39ZCI/FNEtrq5\n/qaIDHeXvZD0+/OIyNUistr9vd0rIoPcZc3fofNF5FP3u//jLv/n9gZV7bc/wCrgmBSv7wtsAj4D\neIHz3XWD7vLTgVE4fyS/BNQDI91lc4Afu8tCwCFJ21Vg1w7iuRr4FPgfYCogScsCwGrgCsAPfBGI\nAr9wl38FeLnN9lr2BxzhbtMD7ANsBE5xl413170XGACUAKOBrcBJ7nuOdZ/v5K6zHdjDff9IYO9c\n/38W008HuXkUsAXYDwgCfwLmu8uGuf8vX8A5Sv+WmyNfa5sjwPHAIqACEGCvpBy+uzmvUsUDfB9Y\nDOzhvncaMDRFrM155XOfHwA0AKe6zz1uDD9x83sisBI43l1+PfAiMBgYA7wHrG0T0zvAWDdnO9ve\nq8C57uMy4AD38deB/wKlON/3GcBAd9kLSb+/rwIfudstA/4N/KPNZ/2rG8s0oAnYK+e5lOsA8vSL\ndBvw8zavfQAcnmY77wCz3cf3AncAY1Ks11mR9wLfAF5xE2QdcL677DD3eXLhX0CGRT7Fvm4CbnQf\nNyfoxKTlP2hO4KTXnsT5gzcAqAZOA0py/f9YjD8d5ObfgN8mPS/DKeTjgfOAV5OWCbCG1EX+KOBD\nnMLrabOPu+m4yH/QnO+dfIbmvKoGGt3Hv2/OYZxG1Kdt3vND4O/u45YC7T7/Gu2L/FeTnne2vfnA\nT4Fhbdb5qvtd2ifFZ3gh6ff3LPA/Scv2cH/3vqTPOiZp+RvAmbnOJeuuSW0c8F330K1aRKpxWguj\nAETkPNnRlVMNTMFpRQFcifPlekOcEQlfzXSnqhpX1VtV9WCcFtYvgbtEZC9335XqZo9rdabbFpHP\niMjz7mFsDXBJUszN1rT5HZze5ndwCE5rrx7nCOYSYL2IPCoie2Yai+mRUST9v6tqHc4R1mh32Zqk\nZQqsbbsBd9lzwC3ArcAmEblDRAZmGMNY4OMuxDwM54/Rd3GOKP3u6+OAUW1y7EfAcHd5q8/T5nGq\n1zrb3oXA7sD7bpfM59zX/4HTgLlfRNaJyG9FxE97rX737mNf0vYBNiQ9bnA/d05ZkU9tDfBLVa1I\n+ilV1TkiMg7nkOwynEPUCmAJTmFHVTeo6kWqOgrnMPDP0sGImnRUtVFVbwWqgMk4J2JHi4gkrbZL\n0uN6nMNNAERkRJtN3gfMA8aq6iDgL80xJ++2ze/gH21+BwNU9Xo3vidV9Vicrpr33d+J6X3rcIoZ\nACIyABgKVOLkyJikZZL8vC1V/aOqzsDJr91xumGgdR6ksgaY1JWg3QbMDUAYpzuyeTuftMmxclU9\nyV3e6vPg/HFpt+k2caXdnqquUNWzgJ2B3wBzRWSAqkZV9aeqOhk4CPgczlFRW61+9zjfvxhO12fe\nsiIPfvfES/OPD6dgXeK2fkVEBohz4rIcp6tCgc0AInIBTkse9/npItKcmFXuugn3+Uac/ryUROTb\nInKEe8LJJ8743HLgbZz+xBjwTRHxi8gXcEbhNHsX2FtEpotICLiuzebLgW2qGhaRWcDZnfxe/gmc\nLCLHuyfaQm5sY0RkuIjMdgtME1CX9BlN9qTKzTnABe7/cxD4FfC6qq4CHgWmisgp7rrfANr+sQdA\nRPZ389uP00AIk2GeAncCPxeR3dzvxz4iMjTDz3Q9zkndEE53Rq178rTEzbMpIrK/u+6DwA9FZLCI\njMZpWHWkw+2JyJdFZCdVTeB0IQEkRORIEZkqIl6ccxpRUufzHOAKEZkgImU4v/sHNM9H9ViRh8dw\n+gubf65TZxTARTiHs1U4J1u+AqCqy4A/4BTdjTgnM19J2t7+wOsiUofTcv6Wqq50l10H3OMeSp6R\nIpYGd9sbcE6ufQM4TVVXqmoE54TaV4BtON0l/25+o6p+CPwMeAZYAbQaaYPTevqZiNTinJh6sKNf\niqquAWbjHO5uxmklfR8nZzzAd3BaNtuAw4FLO9qe6ZZUufkMcA3wL5yW7iTgTABV3YIzKOC3OF04\nk4GFOH+I2xqI05ipwul22Ar8zl32N2Cym6cPp3jvDTj58xROUfwbzsnGTDzq7vMiVY3jtJqnA5/g\n5PydwCB33Z/hdDd9gpPXc9N8FsA5WuhkeycAS93v5s04/eWNOH8I57qfZTnOyd5/pNjFXe7r893t\nh4HLM/zcOdN8AsQUIBG5G+dE1NW5jsXkHxHx4BTJc1T1+VzH01MicilOYT4817EUEmvJG1NE3O61\nCrcr50c4511ey3FY3SIiI0XkYHHGp++Bc+L2oVzHVWgyLvIicpc4FwAsSXrtOhGpFGekyTsiclJH\n2zAmHxVZbh+IM/JlC3AyzrUQjbkNqdsCwO1ALc4Fev8B/pzTiApQxt01InIYzgm2e1V1ivvadUCd\nqv6+1yI0ppdZbptilnFLXlXn45xkM6aoWG6bYpaNPvnLReQ995B3cBa2Z0y+sNw2Ba9Lo2tEZDzw\nSNIh7XCcvj8Ffo5zNWTKKzzFmfjoYoDS0tIZEyZ2+fqgghLVKIGATfKZC0veXrxFVXfqynuyltsD\nSmdM3L1L1wkZk7Fu5XZPinymy9qaMnWazn34yYz3W4jWxyoZPybT60NMNk0sm7BIVWd25T3Zyu2p\n++2j816a15VdG5Ox7uR2j7prRGRk0tNTcS7vN6bgWW6bYpFxf4KIzMGZXGiYOHM6XwscISLTcQ5p\nV+HM1WJMQbHcNsUs4yLvTuzT1t+yGIsxOWG5bYqZXfFqjDFFzIp8L1gfq8x1CMYYA1iR7zU2ssYY\nkw+syBtjTBGzIm+MMUXMirwxxhQxK/LGGFPErMgbY0wRsyJvjDFFzIq8McYUMSvyxhhTxKzIZ5ld\n7WqMySdW5LOoucDb1a7GmHxhRT7LrMAbY/KJFfl8FK+C6ApI1Oc6EmOyRxViqyH2ifPY9Am7CWk+\n0QhaezNE3wT8QAxCn0VKzwWRXEdnTPfFVqLbrwetc55LCCn/Pvj3ym1c/YC15POI1v8VoguBGNAI\nRCE8D629ERLhHEdnTDclGtGan4BuAcLOj1aj269Bm97IdXRFz4p8vtAoNL0IRNssSED0ZbTmW5Co\nzUVkxvRM9HUgkWJBAup+j9b/va8j6lesyOcL7aSlnqhCG+f2TSzGZFOihvaNl2YxCD8B8TV9GVG/\nYkU+X0gZSEUHK8Sg6dU+C8eYrPFNpuPTfwqRt/sqmn7Hiny+EEHKLsE54ZpunWCfhWNM1vh3A/90\nwJtmBS9IqC8j6lesyGdJVq50DeyHDPoVUJ5iYRBCJ/R8H8bkgJR/H0rOSbNUIXBAn8bTn1iRz4Ks\nXunqm4RU3AiekUDI/fFDYBYSOrHn2zcmF8SDlJ4CZd8BAuzI7SBS9j3wDMxtfEXMxslnSVavdPUO\nQSpugdhySGwF3yTwjsre9o3JEQkeAv4ZEH3HecE/HTwluQ2qyFmRz1ci4J+c6yiMyT5PCQQPzHUU\n/YZ11xhjTBGzIl9MEo3OvDc2L4gpJqpOXicacx1JQbLummKQqEPr/gRRd6yxVCBll0Ngam7jMqan\nIovRultAq5zn/n2RssvAk2oEmknFWvJFQGt/7hb4mPOjW9DaX0HcbmBiCli8Eq39JehmWnI7+jZa\n+4tcR1ZQMi7yInKXiGwSkSVJrw0RkadFZIX77+DeCdOkFfvU+SHWZkEUbXw0FxEVHMvt/KSNj9E+\nr2PudMWrcxFSQepKS/5uoO3VOFcBz6rqbsCz7nPTl+KbSP3fmLCWfObuxnI7/8QrST2xmRfim/s6\nmoKVcZFX1fnAtjYvzwbucR/fA5ySpbgKxvpYZW7vBuWbQOrJn/zg37uvoylIltt5yr83qaf5iLp5\nbzLR0z754aq63n28ARjew+2ZrvIOheCRQPK8Nh6gBLFpEHrCcjvHnPwtoXWZCjr57rXbbGYqa6Nr\nVFVFJO3YPRG5GLgYYNSo0dnarQFkwCWodzyEHwFtdObAKTnLLhXPki7l9li7MjlrPOVIxR/Qxvsh\nsgikBEKfs8ZLF/W0yG8UkZGqul5ERgKb0q2oqncAdwBMmTrNBnJnkwhSciKU2Nw2WdSt3J663z6W\n29nkHYqUfSPXURS0nnbXzAPOdx+fD/ynh9sz6ahC5B20/l60YR7Eq3MdUbGz3O4riXoIP4HW3+Pc\nM0HjuY6oqGTckheROcARwDARWQtcC1wPPCgiFwKrgTN6I8h+T2Po9p9DbAXOPTL9aOMcpPxHdsFT\nFlhu51BsFVpzNRAHmlBC4N0ZGfhrm7gsSzIu8qp6VppFR2cpFpOGhp+H2IdAk/uKM5pGa3+HDLkb\nxK5p6wnL7dzR2huBhqRXwhBfhzbORQacm6uwiopVh0IQeZ4dBT5ZHVrzQ4ivT7HMmDwXr4JEqtyN\nQfghtP4+67rJAivyBaGD/6b4CrT6B5Co7btwjMkGSXc7QFd4Hlp3a9/EUsSsyBcACR1H63HwbUXQ\n8NN9FY4x2eEZCN7xgKRZIQKRV5wWv+k2K/I9kJX7umYicCgEZpH+FEoEYh/3TSzGZJGUfwdkEOkL\nvT9Nl47JlBX5bsrqfV07I4KUXwFll5O60AfAN7H34zAm27wjkMG3g29KmhUi7v2OTXdZke+Bvp6z\nRoKHgm9X2hd6PxI6tk9jMSZrxI+UfZ32XZIBCBwIXpsAtCesyBcYKb/GnavGDwj4piAV19sUBqaw\neUchA38G3knuCyEIfda5+Y3pEbszVKHxlCBll0LZpc5VsJKuL9OYAuPfDan4neV1lllLvpDZFyHv\nRCJtb3JhuszyOqusyBuTZavWbs11CMa0sCJvTBb5JdVNLozJHSvyxhhTxKzId0OfXQRlCtJI32jr\nsjF5w4p8N+X0vq4m71mhN/nCirwxvcQKvckHVuSNMaaIWZE3ppdZa97kkhV5Y3rRSN9owAq9yR0r\n8sb0Miv0JpesyBvTB6zQm1yxIm9MH2ku9Mb0JSvyxhhTxKzIG9PHrMvG9CUr8l20PlZpV7uabrMu\nG9PXrMh3gRV4ky3Wmjd9xYq8MX3MRtqYvmRF3pgcsEJv+ooVeWNyxPrnTV+wIm+MMUXMl42NiMgq\noBaIAzFVnZmN7RqTa32R26vWbrUT+qbXZKXIu45U1S1Z3J4x+aLXcnukbzTrY5VW6E2vse4aY3LM\nTsKa3pStIq/AMyKySEQuztI284rd17Xf6pPctpOwprdkq7vmEFWtFJGdgadF5H1VnZ+8gvsFuRhg\n1KjCTGg7nO6X+jS3rdvGZFtWWvKqWun+uwl4CJiVYp07VHWmqs4cPMSS2BSGvsxt67YxvaHHRV5E\nBohIefNj4DhgSU+3a0yu5SK3rdCbbMtGd81w4CERad7efar6RBa2a0yu5SS3m0fcGJMNPS7yqroS\nmJaFWIzJK5bbphjYEEpj8pR12ZhssCJvTB6yvnmTLVbkjclTNnbeZIMVeWPynLXmTU9YkTcmj1m3\njekpK/IZsNv+mVyyQm96wop8J6zAm3xg/fOmu6zIG2NMEbMib0wBsS4b01VW5I0pENY3b7rDirwx\nBcQKvemqbN7+L6/UVdfz+mNvsXXtViZOG8/0o6bg8xftxzX9SORDZcFbC6gI+pl53DTG7DEq1yGZ\nPFaUVW/N+5Xc9PU7iMfjRJuivPrIWzx257Nc+fdvECoL9c5ONQGJ9UApeAf3zj5Mv/fwLU/wwgMv\nI2NjeOrCvPDAKxx3/hGcdNExvbfTRD3oNvAMBwn03n5MryjKIv/3a+YQbgi3PI80NrG1chuP3/Uc\np37zpOztSBVtehYaHgTdAngBAd+eSPn3wDMwe/sy/d66jzbwwv0vE22Kwgrw7xpEa8M8ec8LzDx+\nOjvvMix7O4tXoQ0PQORFoAmnVHih5ItIyRfAmX7ZFICi65Ov2VLL1sqqdq/HojEWPfVul7bV2Zze\n2nAn1P/VLfAAcSAGseXo9l90aV/GdOa9+cuIx+Itz6MfQaI8BIkEi+cvz96OEnVozXch8hROgQeI\nOY8b56JNL2ZvX6bXFV2R93rFufVyCj6/N/MNKRBfy/jhG0Gj7ZfHqyD8NJBiGXGIr4HYmsz3Z0wn\nvD4vIq2/sk6hL8HbldxONEJkiZOjKWj4KdC6NG9ugsZ/Zb4vk3NF111TNriMsXuOYtXSNajuqPb+\noJ+DTml3e87UYmuclrg2odtrAYWyy5HggTvWia8E/DgtnFTikNgGjO3eBzGmjX2Pmcqjtz9FvO0C\nhelHTcloG9r4H2iYg9O1mADvSKT8avAO2bFSdDHp8xo3r02hKLqWPMAFvzybQcMGEiwN4Q/4CIQC\n7LrvBI46+5CO35jYhtbdgdZ8G3QzEAEagTDU3QzxpO4bz1Ag0cHGYqjXLkU32TNs1BBO//7slpwO\nhAL4Az5O+OpRVDelOqJMEnkXrf4ONNzDjrxugvinaO2vW6/rHdHxtqS0B5/C9LWia8kDDB01mJ/N\n+wHLFnxI1cYaxu09hnGTx3T8pkQVWn0FaD2p+3viaPhpZMBXnKe+8eAdA/GP02zQi9DY7c9gTCoH\nnzKLfQ6bzOKXliMiTD10T8oGl7E+VsmqtVtTzrOk4eecc0ct/evJEk63TXw9eEcCICUnOQMK0rbm\nu9A1ZHKuKIs8OP2XUw/bK+P1tfFh0EbSt87jEG9zAUrpl6H2p2nWFxAbSmmyr3xIGQfN3r/Va2lv\n/q0xqP87qQt8kkRtS5HHOxZ8e0MszUAFr3VBFpKiLfIA7724jEfveJqt66sYu8doZl92AuP3TpOg\nkbfosB8SnFa76o7hYx2dgAocBJ6ybsVtTEeaGiM8fuezvP7YW6gqM4+bzue+fgwE26+r8eZux45E\n0NgaxL+78zReBbGladb1IqWn9SB609eKsk8e4NV5C7nr6vtY++E6Gmsb+XDhR9x0ye2sWppiREH0\nA0is63yjiSqIfbDjeWxlmhUFSs/tVtzGdEQTyk2X3M4L97/M9i3bqd1ay0tzX+X3X72N4TKy/XQH\njf8m9QiwNhrn7HgcX4UzqCAFGQn+PbsZvcmFoizymlAe+tNjRMOtkzsajvLwLY+3X7/uFtKOu2y9\nJsQ/3fHUMyTNegHELoQyveCDhR+z4ZPNRCM7jjpj0Rjb1lex+KXljPSN3lHoY6sg8lJmG9ZtTtcO\ngHcYtB/DAwgErMAXmqIs8u88v4SGmoaUy9a+36bFnqiFxMZ2620g1fQHUTS+AdT5AkjpGbQ/Rg5A\n6HiQNC0hY7op3NDEC3NeJtLYvn+9qbGJT5c7ffIthT76DqmLdSo+iK92HnrHgm8i7Xtz/Ujo5O6G\nb3Kk6Ir8f255gnuue6DVGPlkg0cMavNK+2LcXODHjahus0Qh/Bi6/VdO33zwEOfkKyU4XwgfBI9G\nrKvGZFnD9kZ+ddZNLHvtw5TLA6EAw0a3ObKUEjIfCRNDa36MNr3uvLX8R+Dfl5a8lgqk/Crw2UnX\nQlNUJ163rqvi+TkvtTqUbeug2W0uiPKEnGSOLiJ5ZE37At8sArGl6PZrIFaJc7K2eaikD5qeh8AM\nCOznvBRbgTa9BniQ4KHg26V7H870a8/d9xLVm7a3mtYgWSwaZ99jprZ6bdWmvRg3oCt7iUDdH9HG\nERDfjJPb7ndJ69CG+xDfJPCUO1eBR15Box+DdxQSPBw8Nn4+HxVVS37Jy+8Ti3Z0gRJ8uDDFuPaS\nU0k5NCGtCMSWATVAfdLrMSCM1v4W4tVo/d/Rmmsg/BCEH0JrrkQbHu7CfoxxvPH4W8Rj6RsvXr+H\nVUt2DCoY6RsNUsrqqq5OyNcI8U+AOiCc9HoM4h+jtTdBYjtafTladwc0PQoN96JVl0B8bRf3ZfpC\nQbfkNaG88cTbLHj4DRIJpXpjDaodF/mV765q/UJ0KWz/BZ0PM+uKOBp+GMJPJm034TxunAPBg8G7\nUxb3Z4pNzZZanrtvPh+8uZKKYeVsW5/uyNIRDUdZ+e4q9py1a8trI8IPsSGykNUbKjo4Mu0Khdhi\ntP7vkNjKjv7+JqAJrbsVGfTrDt5vcqGgi/zffnwfS19+n0g48wJdPqS81XOtu41OLxTpsjjEVpB6\n6JpAZCGUnJjlfZpiUbWxml+dczNN9RHisRiZTHMXCAUYtFPS+abYaoi8wggiaQYR9EDkDVKe0I2t\nAG0C6cpRseltWemuEZETROQDEflIRK7KxjY7s3rZWpZ0scAHQgGO/8qRO15IhNHEhl6IzgueUaT+\n9QqIXRZeKHKR24/e8QzhunCH3TNtebweZhy7o09eI4vRjIYFd5GUkb5tKBRZD3BR6PH/iIh4gVuB\nE4HJwFkiMrmn2+3Mirc+IZHmJFQqHp+XY849nP1PmA7Auo838PsL/0q0qRe+CL5pUDKb1L/eBPg/\nk/19mqzLVW6//9oKEvGOux2biQjlQ8v51m0XERrgtNjn/9+r3Hf9MzQ1OH8kRhBm9YaKrMQmZd+E\n0JG0L/Qe8O1jQ4fzUDb+7M4CPlLVlaoaAe4HZmdhux0qHzygS/ds9Yiw/PUVqCp11fXc8LXb+GTJ\nWt58tpxIU+u73CQSPqq2jSTNKMyOBQ5FBv4Y8Y2B0vNwhmgG3R8/lF0O3rbDOE2eyklulw3OfEiM\nqhKuC7Nh1WYAFsx7k3//8THefC5AIunvRHOhj8d9NDSUdSO3Bcp/BoHpSMmZ7jj6EE5+h8AzDCm7\nrKsbNX0gG0V+NLTqNlzrvtarph25N+Jpfwsyn9/HTmPb3wYtFo3xyXuruP179/LSv14jFnWOAub+\nZWeWLyol0iQ01nuIRYX3FoS45uwBPHl/BZkfMZfAgCuQ8ita5raRks8iFbchAy5ABnwNGfxXJNjJ\ndMcmn+Qkt4/58mEEQq3vper1edlp7DD8ofb3WI02Rbn32gdY+NQ7PHb7M0TDEaJNwp+uGkP1Fi/h\nBiHc4KG8NsINP53IlV8YQV1NF776vj2Qir8gAXfOek8IGfhrZODVSOn5SPn3kIo/272N81SfdaCJ\nyMUislBEFlZt29r5GzoRKg3yzT9f5MwbXxIkWBqkrGIAl/3pQg747Iy071vy8nJefGCBc59MIBoR\n5ty8M0teH4DHq0SaIB6DYEh5Zu5Qqrf4aAqn3VxzNFB2JRI6tP0i7xAIHQeho+2er0Uq27k98/jp\nHP3lQ/EHfITKnHsiTJo+gavuvZxgKHV3iKryj5/+H9Wba1peW/NRiH/+YTjVW3x4vAnqajwEQ3FQ\n4Z9/GEFT2Mn1jpU5I2bajgYTAf9kKDnJuSZErC8+X2VjdE0lrW9/NMZ9rRVVvQO4A2DK1GlZ6Qgf\nN3kMv3z0R6z9cB2JhDJ2z1F4PB42t52kqXUcNNQ24g/4iEZi+PzK9/+4hoqhMbw+52rXaQfXMWO/\nzfzu8rFcf+k4Dj6phqkH1DFhSileSXVXHA8S6PWuWtP3cpPbAp/7+nEcffZhrFu5gYqdBjF0lNNK\n9qcp8gCxSAyP14vGnaPU3ac3cNF16wkGnZCGj4lx4jlb8QcGs/TpAfz+W7tw5KlVTJisjNwlQvth\nxD4IHtbjj2NyKxt/ft8EdhORCSISAM4E5mVhuxkRjzB2z9GMmzwGj8f5OHsf3PEkSl6fF/E6I1z2\nPbSWAeVxvEl/7iaNrWb4mAi7TQsTbvDw7NzB3HbtrlRuvBaCxwIBnL+PISCIlP/ATjgVp5zmdkl5\niEnTxrcUeIA99t+1g3dAIh7H63Nye/YFW1oKfLNdgmEOn10NKOtXBbnvxhH8+Zq9kIpbQYbh5LTX\n+dc7Bik9J7sfyvS5HrfkVTUmIpcBT+Jkx12qmm4y6j4xaGg5g4YNomZLTcrliUSiZdLJMZOaCJW2\nb3x5vcroCWFWvFvivCeujJg4HCm5FEInQfRd5zZogQPB06Vrx02ByMfcPuKMg5x55NOMvvGH/Gjc\nyecRu6QeXhwqUYIlSlOjgAi7z5gE3qHI4D8713AkNoB3PPin7bh3gilYWbkYSlUfAx7LxrZ6IlwX\nZv7c13jt0UVs37o95TpenxcBolGnM3Lj2gBNjUKwpHWhj8eEzeuck1y+oI+TLzmOYIl70ss3zvkx\nRS9fcnvth+t55p8vsmzBh2kLvC/gIxHTlvH12zb5GDmu/QV50QiEfaUIYUIDApx08THOAvFB8IBe\n+wwmNwr6itdk4bowvzr7Zmq2bCcWTX02yePzMuPYfXjzyR23NVv0fDmf/8oWfAHF7cEhkRAaGoKs\n/mgnJuzx0MrHAAALdUlEQVSzEydccCRTDs38VoLGZNOSl9/nzqv+2TJYIJVBwway+6xdeefZ91pO\npj5yzzDOu3IDwdCOBkw06uWD/w5lQEUZEw/fmzMuPrZVd5ApPkVT5F948BW2b01f4P1BH1/41skc\netpnePv5pUTdK2Wbwh7+8O1dOOvbG5k0pRG8gic4g6G7XcpvnrLkNzmmcN8v/9Vhgff4vFz7r++x\n/LUVvPfCMppnjnx3QRn3/3FnTrlwK+WD43i8pfgHnsYJl57CCQLrY5XUJhK0v/W3KSZFU+TffWFZ\nh1MMR5tizL1hHg/f+jiBkJ9YU7RlzvnN6/3cds14Zl92LHt+cSIycOe+CtuYDlVvrqF+e+ob4DRL\nxOJcdfwv8Pq9xCKt/xi8+dxA3l0wlOse+jaDBg9zZh5wpb35tykqRTO4Nd1NQpLFY3Ga6sPUV9e3\nWz+RgH2P2dfmlTF5JVQaJN5B46VZJByhsbYx5Xzz4/Yaw6BhrQt8snb3hTVFpSiK/MbVm6nakHok\nTaY8Hnjl4TezFJExPZdIJHjnhaU9nmbsk6VrqNqYeqrhkT7nAl4r9MWr4Iv8gv+8ya/PuZm66roe\nbScaibGqemWWojKmZ2KRGDdedDv3/+ahHm/LH/Cx6dMtaZdboS9uBV3k67c38OBvH+7wpFSmSvby\nMWLCTowfk+Y0lCqrlqzhtUcWsXrpGro3e5kxmXnl4TdY82El0XDPczsWiTF8XPqb1ETCUTa+VMXy\nBR9Su61njSWTfwr6xOv7r63A4/NCBn2WHREEr9/HyWenmHsGCNc3cctld1L50Y6550fvPpLL/3Qh\nwVK7QYLJvjcffzttgRePB01kNhWxP+hn2hFTqNg59cynK95ayW1X3IMIeMcneOLmRzjl8hM58kyb\nSK9YFHRL3uv3ImnOJjXPrZ2J3WdO4rzrTqd8SFnK5XNv+C9rPqgk0hhp+VmzvJJ/3/Rot+I2pjO+\nQOr2l3ThClR/wMeRZx3CedednnJ5JBzltivuoakhTLg+TP3SCJGAj//c8gRrP1jXrbhN/inoIj/5\nwD1Sjqrxh/z403xJ2q0b9LP7rF0ZMjL9mPg3n3yHWKT1qIVYJMYbj7/dtYCNydAhX/hMu+mGAcqG\nlO248joDh59+YMtcNm0tW/BByiZSJODjtf8uzHgfJr8VdJEPhPxc/PvzCIQCBEuCLcX9+POPYPCI\nzO6EE22K8voji9KvoEo8mvoOVLE0rxvTUzOOncaMY/fBH/TjD/oJloYYMKiUc39yOvE0F/y14xEW\nv7Q87eJIONqukRT9yBmOvLmu0/m1TYEo6D55gD1n7cqvn7yaxS8uJxKOMPnA3Rk8ooKRE0dw97UP\ntFzZ2hFPipuPtBBhj5mTeP+Nj1qfbPUIe87aLQufwJj2xCN8+Senc8y5h7PirZWUDy5jyiF74gv4\nGD9lF1a+92mn94AVpGVm1lT2OmC3lOPqpTLI7jMnsWrt1vQDEUzBKOiWfLNQaZD9T5zOwafOamnB\nTz9qCidfchyBUIBQaQif30ewNNju8NQf9HPg7P073P6XfnAKpQNLWuby9of8DBhYwpeu/HxvfBxj\nWoyYsDOHnnYA04+a0tJPf9HvzmW3/Sa05HQgFEjZJaOJBPscnv4+B+VDyvj8ZSfiD/pb+voDJUH2\nOmA3Dt7voN75QKbPFXxLviNHn3Moh33xADZ9upWBQ8uor67nhov/QiwSJxqN4fN5mTBlF4740kFs\nZmPa7ey8yzCu+/f3eXXeQtZ+UMmYPUdz0Of3p3RgSR9+GmMcAwaWcvmtX6NmSy311fXsPG4Yz933\nMo/d8bSzggioctaPT0s7mKDZ0Wcfwu77TeDV/y4k0hhl+tFT2PvAPZxbayaw1nwRkEymA8i2KVOn\n6dyHn+zz/YLTB//u80up3lTDhH3GMWnaeHAna7JkLg4TyyYsUtWZudh3LnN7S+U2Fs9fhsfrZfpR\nUxg0rLzH22ye28a+G/mhO7ld1C35VPxBPzNPmJ7rMIzJumGjh3DkWdkd3948iZm16AtXUfTJ95TN\nxGdMes3THpjC1K9a8ol4gqfufZH5Dy4gXN/E7jMncco3T4QxdjhqCtuWddt46KbHWP76CoIlfg7+\nwmc44YIj8fn71VfcpNCvMuAfP/s/3n5uccvl4otfWsbil5Yx/vgRXHTNFzuc38OYfFVXVcdvzv0T\njbWNqCpNDWEe/+szPPG35zjq7EP4/P8cn5Vib102hanfdNdUbazm7WfeSzkfSOXHG/jdBbdSX9Px\nzRmMyUfz575GNBxpd2GTJhLMf2AB//vzf/V4H9ZlU7j6TZFf//HGtPOBgDNNwavz7FJuU3hWvvdp\n2ruiRaMx3nrmvazNLmnTEReeflPkh40Z2uE0BNFwlMoV6/swImOyY9Suw/H60jdgfEEfW9dV9Xg/\nNu98Yeo3RX7nXYYxadq4tF8GfyjAuMl2SGoKzxFnHIzXn/62lfFIjJ3GZqcv3Qp94ek3RR7g4t+d\nx8zjp7Wbnlg8HoIlAQ743IwcRWZM9w0ZWcG3brso5cABfyjAASfvz4BBpVnbn/XPF5Z+VeSDpUHO\nu+4MfvvctRx62gGUlIXwB/3stu9EfnDvZYTKbJoCU5jG7z2Wn8z9Ht/926VMmjYer89H+eAyTrzw\nKM6wOZb6tX41hLJZaXkJZ151KmdedWrLhVAdzSdvTKGYuM94vnPnpX2yLxtSWRj6VUu+LZuXw5ju\nsb75wtGvizxYgTemu6zQF4Z+X+SNMd1nJ2HzX4+KvIhcJyKVIvKO+3NStgIzJpcst7vGWvP5Kxst\n+RtVdbr781gWtmdMvrDczoB12+Q3664xxvSYFfr8lY0if7mIvCcid4mIjUM0xcRyuwusfz4/dXr7\nPxF5BhiRYtGPgdeALYACPwdGqupX02znYuBi9+kUYEk3Y+5Nw3A+T76xuLpmD1Xt9N53ltt5weLq\nmoxyO1nW7vEqIuOBR1R1SgbrLszVPTg7YnF1TX+Jy3K791hcXdOduHo6umZk0tNTyc8WjDFdZrlt\nikVPpzX4rYhMxzmkXQV8vccRGZMfLLdNUehRkVfVc7v51jt6st9eZHF1TdHGZbndZyyurulyXFnr\nkzfGGJN/bJy8McYUsZwV+Xy6bFxEThCRD0TkIxG5KldxpCIiq0Rksfs7ytlNaN2x4ptEZEnSa0NE\n5GkRWeH+2+djydPElbPcyqe8duPJy9y2vO5WXN3KrVy35HN+2biIeIFbgROBycBZIjI5F7F04Ej3\nd5TLIV13Aye0ee0q4FlV3Q141n3e1+6mfVyQ29zKeV5DQeS25XV6d5OlvM51kc8Hs4CPVHWlqkaA\n+4HZOY4p76jqfGBbm5dnA/e4j+8BTunToEgbl3FYbneiP+R1rot8Plw2PhpYk/R8rftavlDgGRFZ\n5F5ZmU+Gq+p69/EGYHgug2kjl7mVD3kN+Z3bltfd0+Xc6tUiLyLPiMiSFD+zgduAicB0YD3wh96M\npYAdoqrTcQ65vyEih+U6oFTUGaaVL0O1ejW3LK+zwvK667qVW716j1dVPSaT9UTkr8AjvRlLByqB\nsUnPx7iv5QVVrXT/3SQiD+Ecgs/PbVQtNorISFVd714huinXAQGo6sbmx72RWwWS15DHuW153XXd\nzetcjq7Jl8vG3wR2E5EJIhIAzgTm5SiWVkRkgIiUNz8GjiO/Lq+fB5zvPj4f+E8OY2mRy9zKo7yG\nPM1ty+vu6W5u9WpLvhN5cdm4qsZE5DLgScAL3KWqS3MRSwrDgYdEBJz/q/tU9YlcBCIic4AjgGEi\nsha4FrgeeFBELgRWA2fkSVxH5DC38iKvIa9z2/K6e3F1K6/tildjjCliuR5dY4wxphdZkTfGmCJm\nRd4YY4qYFXljjCliVuSNMaaIWZE3xpgiZkXeGGOKmBV5Y4wpYv8P9+232wBZsnAAAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f93ce9d6f60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#最小平方与LR相比缺少鲁棒性\n",
    "x_train, y_train = create_toy_data(add_outliers=True)\n",
    "x1_test, x2_test = np.meshgrid(np.linspace(-5, 15, 100), np.linspace(-5, 15, 100))\n",
    "x_test = np.array([x1_test, x2_test]).reshape(2, -1).T\n",
    "\n",
    "feature = PolynomialFeatures(1)\n",
    "X_train = feature.transform(x_train)\n",
    "X_test = feature.transform(x_test)\n",
    "\n",
    "least_squares = LeastSquaresClassifier()\n",
    "least_squares.fit(X_train, y_train)\n",
    "y_ls = least_squares.classify(X_test)\n",
    "\n",
    "logistic_regressor = LogisticRegressor()\n",
    "logistic_regressor.fit(X_train, y_train)\n",
    "y_lr = logistic_regressor.classify(X_test)\n",
    "\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.scatter(x_train[:, 0], x_train[:, 1], c=y_train)\n",
    "plt.contourf(x1_test, x2_test, y_ls.reshape(100, 100), alpha=0.2, levels=np.linspace(0, 1, 3))\n",
    "plt.xlim(-5, 15)\n",
    "plt.ylim(-5, 15)\n",
    "plt.gca().set_aspect('equal', adjustable='box')\n",
    "plt.title(\"Least Squares\")\n",
    "plt.subplot(1, 2, 2)\n",
    "plt.scatter(x_train[:, 0], x_train[:, 1], c=y_train)\n",
    "plt.contourf(x1_test, x2_test, y_lr.reshape(100, 100), alpha=0.2, levels=np.linspace(0, 1, 3))\n",
    "plt.xlim(-5, 15)\n",
    "plt.ylim(-5, 15)\n",
    "plt.gca().set_aspect('equal', adjustable='box')\n",
    "plt.title(\"Logistic Regression\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1.4 Fisher 线性判别函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看作一个高维的向量投影到1维空间上，主要思想是把分类看作一个1维空间，把高维的数据投影到该空间的过程，一方面让类均值的投影分开得较大，同时让每个类别的内部方差较小，最小化类别的重叠。\n",
    "\n",
    "$$ J(w)=\\frac{w^TS_Bw}{w^TS_Ww}$$\n",
    "$S_B$为类间协方差矩阵 $$S_B=(m_2-m_1)(m_2-m_1)^T$$\n",
    "\n",
    "S_W为类内协方差矩阵$$S_W=\\sum_{n \\in C_1}(x_n-m_1)(x_n-m_1)^T+\\sum_{n \\in C_2}(x_n-m_2)(x_n-m_2)^T$$\n",
    "\n",
    "$J(w)$取得最大值的条件为$（w^TS_Bw)S_Ww=(w^TS_Ww)S_Bw$\n",
    "\n",
    "可以看出$w\\propto {S_W}^{-1}(m_2-m_1)$,如果类内协方差为各项同性的，则w正比于类均值的差。该公式是对于数据向一维投影的方向的一个具体选择。然后投影的数据可以用来构建判别函数，方法为选择一个阈值$y_0$,使得当$y(x)\\ge y_{0}$时把数据点分到$C_1$\n",
    "\n",
    "最⼩平⽅⽅法确定线性判别函数的⽬标是使模型的预测尽可能地与⽬标值接近。相反， Fisher判别准则的⽬标是使输出空间的类别有最⼤的区分度。考察⼀下这两种⽅法之间的关系是很有趣的。对于⼆分类问题， Fisher准则可以看成最⼩平⽅的⼀个特例。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQEAAAD8CAYAAAB3lxGOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VNX9//HXZ/ZskLAmBATBAlLEDa0rKrZWK3W37r9a\nv9W61OpXW/fW1u5f+9XWVq1Ua7XVWveqXxXFrWorigKCuCGCBgGBkD2zf35/3BAyzCQkmUlm+zwf\njz6UO5N7DzXznnPPOfdzRFUxxhQvV7YbYIzJLgsBY4qchYAxRc5CwJgiZyFgTJGzEDCmyFkIGFPk\nLASMKXIWAsYUOU82Llo1bJjW1o4blGtFNILPl5W/pjFZs2zR0o2qOrI3783Kp6O2dhwPPjpv0K63\nNrqGCWOHD9r1jMm2ieU7ru7te4vidqDGU8uquk3ZboYxOakoQsAY0z0LAWOKXFGFgN0SGJOsaEKg\nxlMLWBAYs62iCQGwIDAmlaIKAbAgMGZbRRcCsDUIjDFFGgLGmK2KNgRsAZExjqINgS0sCEyxK+oQ\nsEFCY4o8BMCCwJiiDwGwIDDFzUKggwWBKVYWAl1YEJhilLEQEBG3iCwSkScydc5ssIVEpthksidw\nEfBuBs9njBkEGQkBERkLHAncnonz5QK7JTDFIlM9gd8ClwHxDJ0vq2xswBSTtENAROYAn6vqm9t5\n3zkislBEFm6uz/0PlwWBKRaZ6AnsDxwlIquA+4DZIvK3bd+kqnNVdaaqzqwalh+Vfy0ITDFIOwRU\n9UpVHauqE4CTgedV9fS0W5YjLAhMobN1Ar1gQWAKWUZDQFVfVNU5mTxnrrAgMIXKegJ9YAuJTCGy\nTfpMcYh9irY9CNGPwT0OKT0BPDtmu1U5wXoC/WC3BHkm+iHacBmEX4F4HUT+gzZeCeFl2W5ZTrAQ\n6CMbG8g/2nonEAK0y9Ew2vqnLLUot1gI9IMFQZ6Jrkh9PP4paGxw25KDLAT6yYIgj0hZNy8EsI+A\n/T+QFguCPBE4CvBtc9AHgSNAJBstyikWAmmyIMh9UnIMBL4CeIES55/+A5HSU7LcstxgU4QZUOOp\nZW10TbabYbojgpT9F5ScDPH14BoJropstypnWAiY4uEqA9fEbLci59jtgDFFzkIgQ2xbM5OvLAQy\nyILA5CMLgQyzIDD5xkJgAFgQmHxiITBALAhMvrAQMKbIWQgYU+QsBAaY3RKYXGchMIDsuQKTDywE\nBpgFgcl1FgKDwILA5DILgUFiQWBylYXAILIgMLnIQmCQ2d4FJtdYCBhT5CwEssRuCUyusBDIAhsb\nMLnEQiBL8iYINIa2P4U2XIJu/h7a9gBoKNutMhlkIZBF+RAE2nw9tN0NsVXOFl7tD6KNV9umHQXE\nQiDLcjoIoishsgRnC68tIhD7DMJvZKtVJsMsBHJAzgZB5H0gnuKFIBp9Z7BbYwaIhUCOyMkgcFUC\n7hQveEFGDHZrzACxEMghObeQyDeT5O27ANyI/+BBbkwXqtt/j+k123zEdE+8SOXP0ab/gfg6wAVS\ngVRcAu6hg94cDb4EbX8D3QQyDEpPRQKzB70dhSbtEBCRccDdwGicDeDnqurv0j1vMVtVt4kJY4dn\nuxkOdy1S9TuIrQdi4KrJyiaeGnwJWm8Fwh0H6qF1LooggUMGvT2FJBO3A1HgUlWdBuwDXCAi0zJw\n3qKUk2MDAO7R4B6TvV182+6lMwA6hTuOm3SkHQKqulZV3+r492bgXSDHbm7zS84GQTbpxm6Ob7Ix\ngjRldGBQRCYAuwMLMnneYmRBsA3XqNTHZUT2eicFImMhICLlwEPAxaralOL1c0RkoYgs3Fxvv9i9\nYUHQRenpJM9U+DqOm3RkJARExIsTAPeo6sOp3qOqc1V1pqrOrBqWI4NeecCCwCH+/aH8e+CqBlzg\nGg3l30UCs7LdtLyXidkBAe4A3lXVG9JvktlWjaeWtdE12W5G1ol/P/Dvl+1mFJxMrBPYHzgDWCoi\nizuOXaWqT2bg3MZkXvgttP0BiH0OnslI6Sng2SHbrcqatENAVV8BbGRmEOTU+oE8pcEXoPU2Oqcb\nIwvQxiXI0F+AZ0I2m5Y1tmw4TxTc2ICqU6dg8/lo/bfQ5psg1s00YMauGYfWO0lebxBEi3i9gYVA\nHimkINDW25w6BfF1oI0QfgltuBTiSRNLmRNvIPGx6C4iHwzcdXOchUCeKYggiG2G0AskfiAVaEfb\nB3AoyVXe/WvuYQN33RxnIZCH8j4IYqsAb4oXojCQdQrEB/7ZJK838CMlJ/b9fBqF4Hy08Udo408h\ntCAvVy/aU4R5Kq+nDd0jcB452ZbLeT5hAEnZWShxCL3oXA8XlJ4G/n37diKNo03XQfRDtvRotGU5\n+Gcj5WdnuNUDy0LADD73OPBMhOgKEsPAi5TMydx1NAbhBWh4gfMIdODL4JmAlJ8HpWeCNoFrOEg/\nPgbhNzva3/WWJgSh+VAyB9w1GfpLDDwLAZMVUnE12vIHiLzZcaAKKT/fCYhM0Cja9GOnTiJBQNDQ\nfCg7GwkcCq4SoKT/p4+81XHebbkgstRCwAyOGk8tq+rW5P7agehqNPQqEHeW/3p2BFcZMuRyiLeD\nBp1SZpl8ECj8CkQ/Yus3tQJhaJ0Lvv06QiANMhTn47PtbY2AVKR37kFmA4N5zgmC3B0g1LYH0cbL\nIfgQBB9BG69EW+/Z+gZXCbirMv4koAZfJfV0oAeiy9M+v1PRKNXHx91Rli1/WAgUgJwNgtg6aH8A\nZ3GO0vltHHwcop8O7LVdpd28oCCB9M/vHoWUX4pzS1ECBJxbmqE/AUk185G77HYgUxSa6pvx+ryU\nVGTgl6yPcvLWIPw6zgd/W1EILwBPhu7/UxD/V9Dw6yT1BsQPnp0zcxH/XojvLx0zBF7w7JSXtQ0s\nBDJg5durufva+9m8vgFV+MIeO3LmT0+mYlgPi1MGQO4FgZvUj5W4SF3KPIN806HkuI6eyJZfcw8y\n5EcgGewAixe8+V1Nz24H0rR5fQO/v+B2NtRtJBqJEotG+eDNlfz23LmpvwQHWE7dGvj26fYldY91\nFtfENgzY5aX0RKTyNqT8PKTi+8iwPzuDkiaB9QTS9MpDC4hHE3fpicdibF7XwEdvr2LSrhOy07Bc\n4B4OZedC6x9xvm8UiINUQssNKC4gAv4DkbILMvsN3dmGKnAfmPnzFhALgTSt/3Qj0Wiq1W+weW0D\n7DrIDeqJ6qDfs0rgEPDuDpHXnSaEXoHou0CXDU1D/0bdk5CSrw1q24zDbgfStNPuO+INJO/SE4/F\nGLdz9ooud94SqKLt/0Q3nYnWH49u/h6EF/f8w5nmroTAYeA7AKLvkRAAAIQg+MTgtsl0shBI0z5z\n9qR8aCluz9aBLm/Ay/RZ0xg9fmRW2tT1ASNtvx/a7gM6HtGN16HNv4JI+nPlqEJ4Kdp6HwSfhnjz\ndt4fotv6M5pq9Z0ZDBYCaQqU+rn87gvZ75i9GTK8gpFjh/P1c7/KWT87JavtqvHUgsZY/fHzJC+a\nCaNtf0/vAhpFm36CNv8SgvejrX9BN38HIu91/zOuSmc8IEn+LbApJDYmkAEVw8o5+fJjOPnyY7Ld\nlAQ17lLWoqxeV8n46obEF2N1aZ1bg/M7uvZbqvQ4/9Tm65Gq21OPPYgg5ReizT/HWW4bw9nhuAwp\nyW5oFjPrCRQyqaRanEHL1eu2+QZ2j03v3KHnSS7TBWh7R72AbvimI5U3QOBw8OwOJSchlTc5o/gm\nK6wnUMjEA4FjqG5/hHVIlx6BDyk9Nc1z9zTLsJ0ZCHcNUvZf6V3fZIz1BAqclHwDSk+jWvwArF4/\nBam4ErxpLp31fRnwp7hgObjHp3duM6isJ1DoBKdQR8kcasCpRuRLf1mxBGajkYUQeRvn/t4LCDLk\nB1t7CarOuEFkuTMomIlHeLvSLYuPBngJcoGzEDD9I26k4gqIftDxIR8Kvn23fsg1hjb9ouOx3Qjg\ng9Y/I0N/DJ4vpHdtjTmzG8EngSC4xiJl5zjPC5g+s9uBQqQQDkbQeOqHFzL2bIEIeKdA6bEQmJ3w\nLe/MHizHmZ6M41ThaUebrk+7GKe23taxuKhjbUG8Dm3+GUQ/Tuu8xcp6AgXm9acW8chNT9Jc30Kg\n1MdhZx7CYf/v4M6xui0FSgd8N6PQc6Qs6qEtEPu0/9t+xZs7ioRuu1Q7jLY96NyOmD6xnkABWfLi\nO9z7i4do2tiExuO0twR58vbnePrO5xPel/2S5Wn0BOIbSV2uHIh90v/zFjELgQLy+B+fIRKMJByL\nBMM8e/dLxOOJTzoOeBD4DyG5vj8gpeDuoRegEQj9B4LzncpE23KNJnW5cnEqGJs+sxAoIPVrG1Ie\nj4SihFqTF/YMZBBI4CvgmQpsqbLkBwJIxWXdrzGIrkDrz0Jbfo+23oE2XIS23pk4huAqdR5GSgoY\nH1JyQsb/HsXAxgQKSPWOI1n9TnLtvkC5n0BZijl9BnCMQDzIkGshssx5dNhVCb79wVWW+v0ad2YT\naE08HnwGvDPAt+fWU5eehcowCD4G2gqeSUjpWQNarqyQWU+ggBzz3SPw+hPvl70BH0ed/1XE1f0q\nvgHrEYiAbxco/Ybz7d1dAABE3+/mScIQGnw26bxSeiwy7E5k+P3I0F+CN81pxyJmIVBAJs+cxHk3\nnsm4qWPxBryMHDeC0685ngOO/dJ2f3ZLEAwYjUNoIdp6J9r2qLMpacLrEbp/zLibnYRNRtjtQJas\neudTHr/ladZ8uI4RtcM48juHsfM+6X+bTdlrJ67464UZaGEGaQRtuhaiq3Dm9j1o+z+Qiqu3LvDx\nTiX1rIEfCczq33Vja9D2h51diNw7ICXH939qsoBZCGTByrdXc9MFf+ocyW/e3MLNF95B9aTRHHXe\nV5kxa9p2n8Hpq42f1fPyg6+xfvUGvrDHRPY/ei8C5cml0Qdi/YAGn+nYDmzL4GQUiKIt/4tU3eHU\nFhQflH8XWm7CecQ4BgScMt6+ftQIjH6ENl6Ds1oxDrFP0PDrTrXhdJ+bKDAZuR0QkcNF5H0RWSEi\nV2TinIXs4d89mTSVpyhrP1rHn6+5l8dufTqj11uxaBU/P+lGXvj7qyz913Iev3Ue1534vzRtSqwE\nNGBjA6EXSf3YcQhiqzv/KP79kMobIXAM+L+ClP83MuTH/dowVFvvYOtqRXB6GSG0dW6fz1Xo0g4B\nEXEDNwNHANOAU0QkvwuxD7C6Dz7r9rVIMMJz97xMc31LZi6m8Nef3E84GCbWURA1EorQvLmV/7tt\nftLb0w4CVadwaON1aOO1TgB0+4CPkrT/gLsGKTvN2TnYv1f/KxBHV6Q+HlvtjE+YTpnoCewNrFDV\nlaoaBu4Djs7AeQtW5YghPb7u8XpY+fbqHt/TW42bmmnY0Jh0PB6LseRf76T8mXSCQFv+gLb8HqKL\nIboUbbmtY2Av1WPHQzK3C3GS7rYhC5Dxe608l4kQqAW6Tk7XdRwz3Tji24fiS1GheAuNK+VVPUyn\n9YE/4O32iy9QknrtAPRztiD6MYS33Qg05Kz883wBZ4GPB+eDWI4MuWLgSqCXzCHVgiICh+flVmED\nadAGBkXkHOAcgDFjijsjvnTkHrQ0tvHErfMIBxPvlUWE8qoyJs2YkJFrBcoDTJk5ifcXriAW3Vrq\n2xvwMevEfTNyjU6RpWy9B+8qBJ7JSNm3ujx2vLczGDhApOQ4NL4RQi/g/JpHwLdf+hWVClAmQmAN\n0LVPN7bjWAJVnQvMBZi+y65Z2KArtxx66gEc/I19eeXh1/nnzU8hIsRicUbUVHHub8/MaI/1m9d9\ng5vOv50Na+pxuYRYJMpuB3+Rg0/eb7s/u93ZgnibU1hEfKiU4PxKbbu234O4hjpbgA3GNmAagfBr\niGs0Wn6Js1uwezS4hw78tfOQaJrPdouIB/gAOBTnw/8GcKqqpr7hxAmBBx+dl9Z1C0k0HKXug7UE\nyvxU7zhqYC6isGr5p9SvbWCHqWMY0ctpwLVRJ89TBYEGn4fW20j8LomRPBPgRypvGZxiorG1aONV\nHeMQEZzdgid0zDIMXM8j10ws3/FNVe1VHfe0ewKqGhWR7wLzcIZ6/9xTAJhkHp+HCdMHeN27wIQv\njmPCF/t2nW6fLYjVQetcnA9a1+lOH1CO0xsQwIVUXDpo1YS1+QbQrgOhMWfNQNsjSNlJg9KGfJOR\nMQFVfRJ4MhPnMoMsXo+23QfhN0FKIDAHCXw1YfAsVRBo8EWStxMDcEH52YhrlPO6Z3K/5vn7Jd6U\nsO5gq4hTIt1CICVbMZiHYtEYS154h/ffWMHQUUPY76i9qBzVj/vdeAva8H3QZiAGuhna7kJjHzvz\n9F0kBYG2kjoEFNGQU3ZssPU4/1/0w1DdshDIM+FghBu+fSvrP9lIuD2Ex+vhmbte4rwbvsmUvXbq\n07k0+HSKD3PIWeBT+g1wJY4DJATBqL3R0Et01vnrFAdvlrZidleCuzZFb8AD/oOy0qR8YE8R5pkX\n//EqlcM+4pLffMD/PrqCH/x+BVN3q+fOa/7ebWHRbkWXkXg/30E83Rbt7FxItH4ceHdha9EQAD8E\njgV3djZiBZDyS3DGJLasgQiAeyxSclzW2pTrrCeQZzbXPceZl9fhDzgf+NoJEc68Yh333eRh7cr1\njNmpuvcnc9UCy0ia29c4uEZ0+2NbegRScQWEX0dDL4PLj/gPBW+WV4x7xiFVt0H4FTS2EfHs5BQk\n6e/y4yJgIZBnDj32484A2MIfUOZ8cx1xX9/+c0rJ19DQcyRO6XmcGoCeCb04gYD/S4h/+/UKBpWr\nBAJfscXBvWTxmGeqRranPD58dIRR4/r4CLC7Fqm4CmQETgVfD3hnIEN+2Ksfz161YpNJ1hPIMy73\ncNCNScdj8Qo8/VkT75tBrPwW6teupLxqGKVDur8N6GrQ9i8wA85CIN+Ungytf6LrQzqKH3f5yf06\n3Yv/eJXHbnZWb8aiMXabvQun//D4pFqFqVgQFAa7HcgzEpgNpd8EqQBcIOVI6WlI4Ig+n2vJi+/w\n6B+eJtQeItQeIhqJsuSFpdz784d7fY6Ex45VO2oFmnxiPYE8JCWHO4/EahDE3+9HY5/+8/NEtnmK\nMRKO8tb8tznp8qMJlCWXH0ulxj2atS23sWr5AsZXb3Km5Mq+48wUxDYD7eCqsUd4c5SFQL4SQHr3\nIe1O44bmlMddbhetje29DgFt/i3VkYWsw8XqdUMZX/0p2nRdx8KdTwE3SAlS/l3w7ZFWm03m2e1A\nEZu0+wQkxbez2+umanQvlyHH6yHyBhCmOmH1YBhiH+M8SBQCbUCb/8d58Agg8iHa9Gu04WKn+lBs\nQ5p/G9NfFgJF7OvnHoavxI+4tv4aeAM+jr94Di53L381YutBtj+I6Iii7U+hoQVo0w8hssDZRDQ0\nH234b4it7ftfwqTNbgeK2KgdRnDFXy/kydvn89GiVQyvqeKrZ83u2/4H7to+DAbGndAI30biAqUY\n0I62/i2zW4vH1jl7GUbexlnSfBhSemIfQqs4WAjkkZaGVv71wH94/40VjBw7nENOOZDaL/RhmXAK\no3YYwZnX9W96EQDXEPAfDKGX2PLBXr2ukvHVqTZH9Tm1BqNLU7ymEMlgGYpYI9pwGc7ehk65cYKP\nobFPnNqGppPdDuSJxo3N/OykG3jmLy+wYtHHvPZ/b3H9WX9g6b/ezXbT0JKz+eyzWbS1+hkZDRMK\nj2P1xoNIrDDscaoLBw6j28d6XT1XYe5Tm4JP46yl6HqtMEQW2W3HNqwnkCeeun0+rY3txGPOY78a\njxMJxrnn5w/yywOu6XHD0QQKrzyygKfueI6mTc2MHj+S4y46kmn79e/5/3g8zi0X38VHi9cSDu6A\niODxezj0iulwyFTGD3sGtA18+zhP8rkqUN9+EP4PiU8w+qHk2H61IaXoh6R8QhIPRFeDuyZz18pz\n1hPIsjUfrmPx88tYt6rn0fFlr77XGQBdBdvCbFq7OcVPQKgtRKg9cR3Ac/e+zEM3PkHD543EY3HW\nrlzP3Mv+yvtvdLNZx3a8/eJyPlq8qrNqsqo6G6j84t8Ew9NZ3XIVUnUrUvZNcFUAIGXngnc3nOcV\nSpx/Bo5C/Af3qw0pecaT+jsuBu4xmbtOAbCeQJYE20LcetGdrH6vDrfbTSwaY8pekzj712fgSfE0\nYFlFKZvXJd9nayxOyTZ7Cm6s28TdP7mfj5c520FMmjGeM679BlXVQ3nq9vlJZc4joQiP3zKPKXf2\nrSgJwFvzlyadD8Dt8bB5YRs1BweSlxW7AsiQKyFWD3FncRGukj5fuycSOAINPkVi5WOPs7ehbUqa\nwHoCWfLgbx5j1TufEglGCLYGiYQivP/6Cp647dmU75992oFJG5a4PW6+sOdEyoZu3W0nHIzwm7Nu\nYeWS1cSjMeLRGCsWfcxvzrqZpo3NRELblgN3rFv1ecrjGlc+/2QjDRuaUr5eUu5PudYAAX+pr+fd\njNzDwPuFjAeAc+4RyNCfgXuS0xg84D/Q2QnZJLCeQBZoXHnj6cVEI4kfyEg4yquPvs4xFyY/B/Cl\nr+3BmhVreen+f+PxeYhFY9TuVMO3fnZKwvsWPb+UcDBC11LyqkqoLcyHb36Mx+dJui7AyHHJTw8u\n/88H3P2T+wm1hojH4oybWsu3f3VaQj3D/Y7ZmwVPvZW0warLJUzZaxKQxQeNPBORyus7pjDdVlik\nGxYCWRCPxxN2A+pq2w9TJ4HjLjqSr5xxEHUfrKVq1BCqJ45OetumunpC7aGk4+FQhPp1DRx25iE8\nfcdzCV14r9/LUecfnvD+zz/ZyNzL7k5oz6p3PuV35/+Jax/4fufmKOOnjeXo8w/nn394CpfHjYjg\ncgkX3PRfeLxbf722BEFW2LqAHlkIZIHb42b8F8exatknCcdFhKl793xfXjGsvMfFPGOnjMFf4k8K\nAp/fS+3kaqbvNxWvz8O8O1+gpaGFEbXDOe7iI5PO+a8HXyMWSSw7pvE4jZ83sXLpaibOGN95/JBT\nDmCvI3bng4Ur8ZV4mbr3TgkBYHKb/ZfKklOuPI4bz/4j0UiUaCSK1+fB4/dy/CVz+nyu9uYgwbYg\nVaMqmX7AVIbVVPH5Jxs7tyL3eD2MGDuMaftOBoHZpx7A7FMPQOPa7dTips/qU85GiEtoTDE+UF5Z\nxh5f3mW7bbXaA7nHQiBLxk6u4YcPXMLLD77Gmg/XMn76OA487kuUV5X3+hytTW3c9aN/8P7rKxCX\nUDaklNN+eALfv+M8Hv/jMyyctwQBZh6xO3PO/QouV+I9cU9rC6bsvRPvLvgw6VHjaCTK+Gl92MVI\nYUPdRjw+DzWjrQhJLkp7L8L+sL0I+y7UHmbBE2+x/LX3GVZdyawT9uVv1z3IJ+/VJe02fPld36Um\nxXhBn67XFuLnJ/+Wxg1NRDt6FL6Aj72O2J1Tr+pd+e6PFq/izmv+TktjG8TjjJ4wirP/5wwio506\niRYEA6cvexFaCOSBYEuQX53xexo2NhIJRhCXC4/HRTxOZ5d/C3G52O/omZx61fFpX7e1qY1n//IS\ni19Yhr/Ux8En7c++X5/Zqx2TGzY08ZPjrk8YgBQRho4Ywk8fv4L16izdtSAYGIO6IakZePPvfZnN\n6xs6p/Y0HicSTr3llsbjfP5JZqoAlw0p5ZjvHcEx3+t76bLXHl9IPLbNwKIqwdYQy1/7gOn7T7Vb\ngxxhE6d5YPH8pSnn9lPx+jxM3nPiALdo+zat3ZyyzbGOGQag54VEZtBYCOSBQHnqMl/icuH1bZ0D\nd7ldBEoDzDpx38FqWrem7DkJX4k/+QVVdpyxddnuliAw2WMhkAcOPmm/pCXD4nIxdnINJ37/KKon\njGboiCHs8/WZXHHP9yivLCMSihDubuFRmjavb+DZu1/knzc/zYpFq1I+Gbzb7OkMr6lKWC/gDfiY\nfuDOjJmUXg0Ek1k2MJgPFB684XFefvg1PF4PGleGDK/golvPpqq6MuGt9Wsb+Ot1D7Bi0ceAMmnX\nCZx+7YmMGDMsI01Z/Pwy/vKj+9CYEo1G8QV8TD9wKmf97NSkKcdgW4j5f/0XC+ctxuv3cOBx+3LA\n8XsnTVVuWUloYwOZY7MDBarh80ZWLfuUISMqmLjL+KRR+mg4yo+O/jVN9S1o3BmUExHKK8v46eNX\n9GpDkZ6E2sNccdhPk54a9AV8nPnTk9n14C/2+9wWBJnVlxCw24E8UjlqKLvNnu4s2U0xTff2S8sJ\ntoY6AwCcEflwMMJbz6Uq6dU3H761MmUB0nAwzBtPLUrr3DZImD0WAgXk8083EQ4ljwOE2kNsqqtP\n+/zbduMTXuttdeIeWBBkR1r/5UTkehF5T0TeFpFHRKRy+z9lBsrYKTX4UnT5/aV+aienX05r8p4T\nU64T8gV87PP1XvU8t8uCYPClG9/PAtNVdQbwAXBl+k0y/TVt38kMr63C7dk6Iu/2eKgcNZRdDtw5\n7fN7fB7Ovv4MfAEfvoAPj9eD1+9l36NmMm2fyWmffwubNhxcaa0YVNVnuvzxNeCE9Jpj0uFyubj0\nT+fx2K3zWDhvCarKnoftytHnH56R7jrAlL124hdPXs3iF5YRbA2y8z6Tqd5xVEbObbIjY7MDIvI4\n8A9V/dv23muzA2Z7bLYgPRl9dkBE5gOpVndcrar/7HjP1TgVHe/p4TznAOcAjBlj3b2BFA5GeP7e\nl3n9qUW43C4OOHZvDjx+H9wed5/PVff+Z7z13Nu4xMX0WTtTPX5ktysYMylrJcmKUNo9ARE5E/gO\ncKiqtvXmZ6wnMHDisTjXf+tm1q5c11lU1BvwMmXmJM678Vt9OtdjtzzN8/e+QjQU6VwUKCKMmVTN\nGdeeyLipAx/m1iPon0FbJyAihwOXAUf1NgBMejbWbeKua//BNV//Jb8565akHYiWvfIe61dtSKgq\nHAlGeH/hSlYvr+v1dT5bsY7n732FSJcAAGfdwZoVa/ntd+bStCn11uaZZLMFAy/d0aI/ABXAsyKy\nWET+mIE2mW5sXFPPr864iTeeXszmdQ18vHQ1d1x9Ly/d/+/O93y4aGXKQqMai7NyyapeX2vJS+8Q\njaQuhgpdChDWAAAIK0lEQVROHYNXH1nQp/b3lwXBwEorBFR1J1Udp6q7dfzv3Ew1zCR76vbnCLaF\nE1YERoJhHrt5HtGw880/bHRlyuXBbq+boSOHJh3vjtvjxtVD+bFIOMq6VRv70Pr0WBAMHFsxmEdW\nLFqZEABbqCob1zgrAvc6YvfkWoKA1+dlxqzerxXY49BdepxW9AV8TJwxuDv52PqBgWEhkEe6bvrR\nVSwao2KYU6C0vLKM791yNsNrqvAGfHj9XqonVvPffzo35fZm3RkxdjjHXTwHj8+TtMOQy+2mpDzA\nl+bs2f+/jMkZVl4sDwRbg7z66BuEO+oLdu0NeH0eph+4c8JWZBOmj+O6f17BhrqNuNxuho+p6td1\nZ52wDzMOmsbiF5bx3n8+YNW7dcQiMWYcNI2jLzicQGmKoiGDwKYNM8seJc5xrU1t/PK039GyuZVI\nKIKIoKrOt7rCjIOncfoPT8Rf4tv+yQqETRtunxUaLSDP3vUSzZtathYZ7Qhtn9/LtQ//gPLKsmw2\nLytsIVFm2ZhAjlvy4rKUBTuj0Tgt9S1ZaFFusNmCzLEQyHElFam37dZYbFCW7+YyC4LMsBDIcYec\ndEDqIqNTarudLSgmFgTpsxDIcXsdvhv7H7s3Hq+HQFkAX4mf6vEj+favT89203KGBUF6bGAw1wmc\ncMnX+fIZB/HJu3UMHTGE8TuP7dVWYMVky2Ch6TsLgTxROXIIlSOnZbsZpgDZ7YAxRc5CwBSMGk+t\njQv0g4VAsRr8haKDwoKg72xMoMgsem4pj9z0JJs+q2fI8AqOPOfLHHDsPgU10OgEwRpbTdhL1hMo\nIm//azl3/fgfbPrMeey4aVMzD934f7z4wL+385P5x3oEvWchUEQeu3kekW12Kg4Hwzw599mCvD2w\nIOgdC4EisvGz1B+I9pYQoW02GS0UFgTbZyFQREZ2c49cWhHAHyieR5FNIguBInL0BUfgDSTWH/QG\nfBz5ncNSDgxqXHnpgf9w3Ym/4aqv/YL7fv0ozUX85GKhshAoItMPmMpZPz+VkTuMQFwuqkZVctIP\njmbWCfukfP89P3uIR256kvWrNtC4oZFXH32dX572O4ItwUFueXrslqBnNkVYZGbMmsaMWdtffly/\ntoE35i3qrGIMEI/GaGtp55VHX+fLp88ayGYOCCtCkpr1BExKn7xbh8eb/B0RCUb48M2PstCi9NiT\nht2zEDApVVVXEo8llzd3edyMHDcyCy1KnwVBahYCJqXxO49l5NjhuLbZxNTtcXPQianHEPKBBUEy\nCwGTmsCFN3+byXtMxOPx4PV5qKqu5LwbvsnIcSOy3bq02CYmiWxg0HSrYlg5F978bVqb2gi3h6ka\nVVlQzxgYh/UEzHaVDSmlanThBYDdEjgsBExRsrGBrSwETNGyIHBYCJiiZkFgIWBM0QeBhYAxFHcQ\nWAgY06FY1w9kJARE5FIRURHJ71UkxhShtENARMYBhwGfpN8cY7Kv2G4JMtETuBG4jIKsUmeKTTGO\nDaQVAiJyNLBGVZdkqD3GZF2xBcF2nx0QkflAdYqXrgauwrkV2C4ROQc4B2DMmOIcgDH5Y8sGp8VQ\niERU+9eLF5FdgOeAto5DY4HPgL1VdV1PPzt9l131wUfn9eu6xgymLTsd51sQTCzf8U1Vndmb9/b7\ndkBVl6rqKFWdoKoTgDpgj+0FgDH5pBhuDWydgDHbUehBkLEQ6OgRbMzU+YzJJYW8kMh6AsYUOQsB\nY/qgEG8JLASM6aVCHRuwEDCmDwoxCCwEjOmjQgsCCwFj+qGQgsBCwJh+KpQgsBAwJg2FEAQWAsak\nKd8XElkIGFPkLASMKXIWAsZkQI2nNm/HBSwEjMmQfA0CCwFjMigfg8BCwJgMy7cgsBAwpshZCBhT\n5PpdaDSti4psAFYPwKlHAPlS3Sif2gr51d58aisMTHvHq+rI3rwxKyEwUERkYW8rrGZbPrUV8qu9\n+dRWyH577XbAmCJnIWBMkSu0EJib7Qb0QT61FfKrvfnUVshyewtqTMAY03eF1hMwxvRRQYaAiFwq\nIioiI7Ldlp6IyPUi8p6IvC0ij4hIZbbbtC0ROVxE3heRFSJyRbbb0xMRGSciL4jIchF5R0Quynab\ntkdE3CKySESeyFYbCi4ERGQczk7Jn2S7Lb3wLDBdVWcAHwBXZrk9CUTEDdwMHAFMA04RkWnZbVWP\nosClqjoN2Ae4IMfbC3AR8G42G1BwIQDcCFwG5Pxgh6o+o6rRjj++hrOzcy7ZG1ihqitVNQzcBxyd\n5TZ1S1XXqupbHf/ejPPhytmyPyIyFjgSuD2b7SioEBCRo4E1qrok223ph7OAp7LdiG3UAp92+XMd\nOfyh6kpEJgC7Awuy25Ie/RbnCyuezUZ4snnx/hCR+UB1ipeuBq7CuRXIGT21V1X/2fGeq3G6svcM\nZtsKlYiUAw8BF6tqU7bbk4qIzAE+V9U3ReTgbLYl70JAVb+c6riI7ALsCCwREXC61m+JyN6qum4Q\nm5igu/ZuISJnAnOAQzX35mvXAOO6/Hlsx7GcJSJenAC4R1UfznZ7erA/cJSIfA0IAENE5G+qevpg\nN6Rg1wmIyCpgZi5vly4ihwM3AAep6oZst2dbIuLBGbA8FOfD/wZwqqq+k9WGdUOc9L8LqFfVi7Pd\nnt7q6Al8X1XnZOP6BTUmkIf+AFQAz4rIYhH5Y7Yb1FXHoOV3gXk4g2z352oAdNgfOAOY3fH/5+KO\nb1rTg4LtCRhjesd6AsYUOQsBY4qchYAxRc5CwJgiZyFgTJGzEDCmyFkIGFPkLASMKXL/Hzt2WMmP\neNq2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f93d2c00550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_train, y_train = create_toy_data()\n",
    "x1_test, x2_test = np.meshgrid(np.linspace(-5, 5, 100), np.linspace(-5, 5, 100))\n",
    "x_test = np.array([x1_test, x2_test]).reshape(2, -1).T\n",
    "feature = PolynomialFeatures(1)\n",
    "X_train = feature.transform(x_train)\n",
    "X_test = feature.transform(x_test)\n",
    "\n",
    "model = LeastSquaresClassifier()\n",
    "model.fit(X_train, y_train)\n",
    "y = model.classify(X_test)\n",
    "\n",
    "plt.scatter(x_train[:, 0], x_train[:, 1], c=y_train)\n",
    "plt.contourf(x1_test, x2_test, y.reshape(100, 100), alpha=0.2, levels=np.linspace(0, 1, 3))\n",
    "plt.xlim(-5, 5)\n",
    "plt.ylim(-5, 5)\n",
    "plt.gca().set_aspect('equal', adjustable='box')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1.5与最小平方的关系 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们让属于C1的⽬标值等于$\\frac{N}{N_1}$ ，其中N1是类别C1的模式的数量， N是总的模式数量。这个\n",
    "⽬标值近似于类别C1的先验概率的导数。对于类别C2，我们令⽬标值等于−  $\\frac{N}{N_2}$，其中N2是类\n",
    "别C2的模式的数量。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "平方误差函数为:$$E=\\frac{1}{2}\\sum_{n=1}^N(w^Tx_n+w_0-t_n)^2$$\n",
    "对$w_0$求导：\n",
    "$$\\sum_{n=1}^{N}(w^Tx_n+w_0-t_n)=0$$\n",
    "对$w$求导：\n",
    "$$\\sum_{n=1}^{N}(w^Tx_n+w_0-t_n)x_n=0$$\n",
    "再由:\n",
    "$$\\sum_{n=1}^Nt_n=N_1\\frac{N}{N_1}-N_2\\frac{N}{N_2}=0$$\n",
    "得到偏置项的表达式:\n",
    "$$w_0=-w^Tm$$\n",
    "m是所有数据的均值:$$m=\\frac{1}{N}\\sum_{n=1}^Nx_n=\\frac{1}{N}(N_1m_1+N_2m_2)$$\n",
    "\n",
    "对w求导的公式就可以写为:$$(\\mathbf{S_W}+\\frac{N_1N_2}{N}\\mathbf{S_B})w=N(m_1-m_2)$$\n",
    "\n",
    "代入$\\mathbf {S_B}$公式，可以得出:$w\\propto {S_W}^{-1}(m_2-m_1)$因此权向量恰好与根据Fisher判别准则得到的结果相\n",
    "同。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1.7 感知器算法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对应于⼀个⼆分类的模型，这个模型中，输⼊向量x⾸先使⽤⼀个固定的⾮线性变换得到⼀个特征向量ϕ(x)，这个特征向量然后被⽤于构造⼀个⼀般的线性模型，形式为：\n",
    "$$y(x)=f(w^T(\\phi(x))$$\n",
    "其中激活函数:$$f(a)=\\begin{cases}+1,& a\\ge 0\\\\\n",
    "-1,& a < 0\\end{cases}$$\n",
    "\n",
    "目标变量:$$t_n\\begin{cases} +1,& t_n\\in \\mathcal C_1\\\\\n",
    "-1,& t_n\\in \\mathcal C_2\\end{cases}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "寻找一个权向量$w$使得对$\\mathcal C_1$中的模式$x_n$都有$w^T\\phi(x_n)>0$,对于类别$\\mathcal C_2$中的$x_n$有$w^T\\phi(x_n)<0$,所有模式满足$w^T\\phi(x_n)t_n>0$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于正确分类的模\n",
    "式，感知器准则赋予零误差，⽽对于误分类的模式$x_n$，它试着最⼩化$-w^T\\phi(x_n)t_n$\n",
    "\n",
    "$$E_p(w)=-\\sum_{n\\in \\mathcal{M}}w^T\\phi_nt_n$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "利用梯度下降优化:$$w^{\\tau+1}=w^{\\tau}+\\eta\\phi_nt_n$$\n",
    "\n",
    "* 如果模式正确分类，那么权向量保持不变\n",
    "* 如果模式被错误分类，那么对于类别C1，我们把向量$\\phi(x_n)$加到当前对于权向量w的估计值上，⽽对于类别C2，我们从w中减掉向量$\\phi(x_n)$"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
